auto_metrics Nightly Notebook

By Josh Dillon

Last Updated January 27, 2021

auto_metrics is a module in hera_qm that computes a series of statistics on night-long autocorrelation waterfalls to find antenna outliers in shape, power, or temporal structure. In general, these are assessed by collapsing each waterfall to a single (normalized) spectrum, then comparing that spectrum to the mean/median of the unflagged antennas' spectra to compute some difference metric. Those are then converted into modified Z-scores by comparing the overall distribution of good antennas and the worst antenna is flagged if it exceeds some threshold. This whole processes is repeated iteratively until no new bad antennas are identified. This proceeds in two rounds, first with more robust median-based statistics to identify the worst outliers, and then (after an RFI flagging step), a round with mean-based statistics. This notebook examines those mean-based spectra and statistics.

Statistics computed (after removing the worst offenders with a median-based metrics and then RFI flagging):

  • Shape: Compute the mean spectrum over time for each autocorrelation, but then divide by the mean of the whole waterfall to get something near 1. Now compute the mean absolute value of the difference between that and the mean spectrum over all non-excluded antennas of the same polarization. Convert this to a modified Z-score by comparing to all non-excluded antennas.
  • Power: Compute the mean spectrum over time for each autocorrelation without normalizing. Now compute the mean absolute value of the difference between the log of that (because we care about power outliers in dB rather than linear units) and the log of the mean spectrum over all non-excluded antennas of the same polarization. Convert this to a modified Z-score by comparing to all non-excluded antennas.
  • Temporal variability: Divide each autocorrelation by the mean waterfall for all non-excluded antennas of the same polarization. Now reduce to a single spectrum by computing the standard deviation along the time axis for each. Now compute the mean value (not the absolute value, since low variability shouldn't get a high Z-score) of the difference between that and the mean spectrum over all non-excluded antennas of the same polarization. Convert this to a modified Z-score by comparing to all non-excluded antennas.
  • Temporal discontinuities: Divide each autocorrelation by the mean waterfall for all non-excluded antennas of the same polarization. Now compute the element-by-element difference along the time axis, take the absolute value, and take the mean along the time axis to get a single spectrum. Now compute the mean value (not the absolute value, since low variability shouldn't get a high z-score) of the difference between that and the mean spectrum over all non-excluded antennas of the same polarization. Convert this to a modified Z-score by comparing to all non-excluded antennas.
In [1]:
import numpy as np
import pandas as pd
pd.set_option('display.max_rows', 1000)
import matplotlib.pyplot as plt
import matplotlib
import glob
import os
import operator
from hera_cal.io import HERAData
from hera_cal.utils import split_pol
from hera_qm.metrics_io import load_metric_file
from hera_notebook_templates.utils import status_colors
from IPython.display import display, HTML

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

Parse Inputs and Load Data

In [2]:
# If you want to run this notebook locally, copy the output of the next cell into the first few lines of this cell.
# JD = '2459122'
# data_path = '/lustre/aoc/projects/hera/H4C/2459122'
# os.environ["JULIANDATE"] = JD
# os.environ["DATA_PATH"] = data_path
In [3]:
# Use environment variables to figure out path to data
JD = os.environ['JULIANDATE']
data_path = os.environ['DATA_PATH']
print(f'JD = "{JD}"')
print(f'data_path = "{data_path}"')
JD = "2459241"
data_path = "/mnt/sn1"
In [4]:
print(f'Looking for data in {data_path} on JD {JD}...')

auto_metrics_file = sorted(glob.glob(os.path.join(data_path, f'zen.{JD}*.auto_metrics.h5')))
if len(auto_metrics_file) > 0:
    auto_metrics_file = auto_metrics_file[0]
    print(f'Found auto_metrics results file at {auto_metrics_file}.')
else:
    raise OSError(f'{auto_metrics_file} not found.')

raw_auto_files = sorted(glob.glob(os.path.join(data_path, f'zen.{JD}.?????.sum.autos.uvh5')))
if len(raw_auto_files) > 0:
    print(f'Found {len(raw_auto_files)} extracted autocorrelation files.')
else: 
    raise OSError(f'No files of the form zen.{JD}.?????.sum.autos.uvh5 found in {data_path}.')
Looking for data in /mnt/sn1 on JD 2459241...
Found auto_metrics results file at /mnt/sn1/zen.2459241.49564.sum.auto_metrics.h5.
Found 600 extracted autocorrelation files.
In [5]:
# load auto_metrics and define use someful quantities
am = load_metric_file(auto_metrics_file)
mean_round_modz_cut = am['parameters']['mean_round_modz_cut']
ex_ants = am['ex_ants']['r2_ex_ants']
ants = sorted(set(bl[0] for bl in am['modzs']['r2_shape_modzs']))

# load raw autocorrelation waterfalls and define some useful quantities
hd = HERAData(raw_auto_files)
autos, _, _ = hd.read(axis='blt')
wf_shape = next(iter(autos.values())).shape
freqs = autos.freqs / 1e6
times = autos.times
lsts = autos.lsts * 12 / np.pi
In [6]:
# try to load a priori antenna statusesm but fail gracefully if this doesn't work.
a_priori_statuses = {ant: 'Not Found' for ant in ants}
try:
    from hera_mc import cm_active
    h = cm_active.ActiveData(at_date=JD)
    h.load_apriori()    
    for ant_name in h.apriori:
        ant = int("".join(filter(str.isdigit, ant_name)))
        if ant in a_priori_statuses:
            a_priori_statuses[ant] = h.apriori[ant_name].status
except Exception as err:
    print(f'Could not load a priori antenna statuses.\nEncountered {type(err)} with message: {err}')

Summary Plots and Tables

In [7]:
# print ex_ants for easy copy-pasting to YAML file
print('ex_ants: [' + ", ".join(str(ant) for ant in ex_ants) + ']')
ex_ants: [0, 1, 2, 11, 12, 13, 14, 23, 24, 25, 26, 39, 73, 82, 83, 84, 85, 86, 87, 92, 93, 94, 99, 102, 103, 112, 116, 121, 122, 123, 127, 128, 130, 136, 138, 143, 145, 155, 156, 157, 158, 165, 166, 178, 179, 184, 186, 187]
In [8]:
def Array_Plot():
    plt.figure(figsize=(8, 8), dpi=75)
    plt.scatter(np.array(list(autos.antpos.values()))[:,0], 
                np.array(list(autos.antpos.values()))[:,1], c='w', s=0)
    for ant, pos in autos.antpos.items():
        bad = ant in ex_ants
        plt.gca().add_artist(plt.Circle(tuple(pos[0:2]), radius=7, 
                                        fill=(~bad), color=['grey','r'][bad]))
        plt.text(pos[0],pos[1],str(ant), va='center', ha='center', color='w')
    plt.xlabel("Antenna East-West Position (meters)")
    plt.ylabel("Antenna North-South Position (meters)")
    plt.title(f'Antenna Positions and Auto_Metrics Flags on {JD}\n(Maximum Modified Z-Score > {mean_round_modz_cut} in Red)');
    plt.axis('equal')
    plt.tight_layout()

Figure 1: Antenna Positions with auto_metrics flags.

This plot shows the antenna positions of all antennas in the data. The antennas with at least one Modified Z-score for one metric on one polarization exceeding the cut are entirely flagged.

In [9]:
Array_Plot()
In [10]:
# Parse modzs for Table 1 and other figures
modzs_to_check = {'Shape': 'r2_shape_modzs', 'Power': 'r2_power_modzs', 
                  'Temporal Variability': 'r2_temp_var_modzs', 'Temporal Discontinuties': 'r2_temp_diff_modzs'}
worst_metrics = []
worst_zs = []
all_modzs = {}
binary_flags = {rationale: [] for rationale in modzs_to_check}

for ant in ants:
    # parse modzs and figure out flag counts
    modzs = {f'{pol} {rationale}': am['modzs'][dict_name][(ant, ant, pol)] 
             for rationale, dict_name in modzs_to_check.items() for pol in autos.pols()}
    for pol in autos.pols():
        for rationale, dict_name in modzs_to_check.items():
            binary_flags[rationale].append(am['modzs'][dict_name][(ant, ant, pol)] > mean_round_modz_cut)
    
    # figure out which metric is the largest outlier
    worst_metric, worst_z = max(modzs.items(), key=operator.itemgetter(1))
    worst_metrics.append(worst_metric)
    worst_zs.append(worst_z)
    
    # parse out all metrics for dataframe
    for k in modzs:
        col_label = k + ' Modified Z-Score'        
        if col_label in all_modzs:
            all_modzs[col_label].append(modzs[k])
        else:
            all_modzs[col_label] = [modzs[k]]
In [11]:
# build dataframe
to_show = {'Ant': ants, 'A Priori Status': [a_priori_statuses[ant] for ant in ants],
           'Worst Metric': worst_metrics, 'Worst Modified Z-Score': worst_zs}
to_show.update(all_modzs)
df = pd.DataFrame(to_show).sort_values('Worst Modified Z-Score', ascending=False)

# style dataframe
z_score_cols = [col for col in df.columns if col not in ['Ant', 'A Priori Status', 'Worst Metric']]
table1 = df.style.hide_index()\
           .applymap(lambda val: 'font-weight: bold' if val in ex_ants else '', subset=['Ant']) \
           .applymap(lambda val: 'color: red' if val in ex_ants else '', subset=['Ant']) \
           .applymap(lambda val: f'background-color: {status_colors[val]}' if val in status_colors else '', subset=['A Priori Status']) \
           .background_gradient(cmap='viridis', vmax=mean_round_modz_cut * 3, vmin=0, axis=None, subset=z_score_cols) \
           .applymap(lambda val: 'font-weight: bold' if val > am['parameters']['mean_round_modz_cut'] else '', subset=z_score_cols) \
           .applymap(lambda val: 'color: red' if val > am['parameters']['mean_round_modz_cut'] else '', subset=z_score_cols) \
           .set_table_styles([dict(selector="th",props=[('max-width', '70pt')])])

Table 1: Modified Z-Score Summary

This table displays the metrics for each antenna, highlighting which one is the worst. It is sorted by each antenna's worst metric. When one metric exceeds the threshold, auto_metrics recommends cutting that antenna. Flagged antennas and metrics exceeding the cut are shown in bold and red. Also shown is the antenna's a priori status.

In [12]:
HTML(table1.render())
Out[12]:
Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
2 RF_maintenance nn Temporal Discontinuties 2114.273409 434.747897 474.186072 nan nan 784.128533 749.796613 2114.273409 1972.402185
1 digital_maintenance nn Temporal Discontinuties 1311.205620 529.299680 533.324560 nan nan 462.830001 433.445162 1311.205620 1011.981816
24 RF_maintenance ee Temporal Discontinuties 1262.808006 531.062547 533.230048 nan nan 257.255477 655.066687 421.164272 1262.808006
0 digital_maintenance nn Temporal Discontinuties 1239.341529 386.076769 331.281266 nan nan 556.289011 510.911822 1239.341529 1150.166483
12 RF_maintenance ee Temporal Discontinuties 1227.818790 458.936889 427.402072 nan nan 399.407473 474.281693 915.141199 1227.818790
23 RF_maintenance nn Temporal Discontinuties 1012.105831 281.294592 371.828143 nan nan 402.627523 421.157687 1012.105831 979.650203
14 digital_maintenance ee Temporal Discontinuties 1005.918902 476.830517 314.974753 nan nan 297.174015 356.167247 621.649728 1005.918902
13 digital_maintenance ee Temporal Discontinuties 987.620856 529.987959 533.390714 nan nan 366.530437 444.024939 817.126921 987.620856
11 digital_maintenance nn Temporal Discontinuties 882.986741 498.613135 490.080818 nan nan 346.541226 250.515666 882.986741 531.133810
39 digital_maintenance ee Temporal Discontinuties 881.276120 531.337695 533.857976 nan nan 134.277709 482.114307 280.391022 881.276120
127 digital_ok nn Temporal Discontinuties 600.595454 39.331387 0.034355 nan nan 508.242101 94.401801 600.595454 119.339639
25 digital_maintenance ee Temporal Discontinuties 556.153810 517.219840 520.209054 nan nan 283.001030 268.928329 534.881701 556.153810
121 RF_maintenance ee Shape 531.953438 122.794166 531.953438 nan nan 21.631198 27.253250 22.589585 27.197906
116 RF_maintenance ee Shape 531.719250 111.547255 531.719250 nan nan 21.310906 30.316771 29.919520 32.308088
73 digital_ok ee Shape 529.755837 4.499949 529.755837 nan nan 7.526088 18.811282 7.242069 25.347927
143 digital_ok ee Shape 528.753541 72.182798 528.753541 nan nan 31.476749 24.021851 5.526966 19.250580
26 digital_maintenance ee Shape 528.537610 523.880316 528.537610 nan nan 273.689379 144.747555 488.313535 210.718513
187 digital_ok ee Shape 528.067346 6.997110 528.067346 nan nan 14.242670 23.986164 15.944763 16.274843
103 digital_ok ee Shape 527.865890 1.862067 527.865890 nan nan 9.076598 12.845035 5.201319 5.543980
184 RF_maintenance ee Shape 525.791105 3.235650 525.791105 nan nan 13.479134 16.120330 8.819277 13.891883
112 RF_maintenance ee Temporal Discontinuties 519.116906 19.491544 39.130278 nan nan 209.422162 462.214718 126.033189 519.116906
158 digital_ok ee Shape 519.009014 9.094384 519.009014 nan nan 20.326965 25.077673 12.460530 16.395519
94 digital_ok ee Shape 515.134757 6.994364 515.134757 nan nan 14.481489 31.901461 14.049587 41.844616
122 digital_maintenance ee Shape 513.241409 7.382939 513.241409 nan nan 10.746671 12.911785 7.195036 15.237563
128 digital_ok ee Shape 508.262728 3.854949 508.262728 nan nan 185.177185 37.270363 129.634712 39.339875
155 RF_maintenance ee Shape 504.018185 122.131750 504.018185 nan nan 30.022377 68.463956 44.094161 65.059389
138 RF_maintenance ee Shape 458.555166 1.779760 458.555166 nan nan 12.251629 11.871141 3.084650 11.362052
178 digital_ok ee Shape 457.487924 3.729029 457.487924 nan nan 12.722815 31.536858 7.177451 16.364962
85 digital_ok nn Temporal Discontinuties 192.084116 11.156367 6.768437 nan nan 101.017974 85.549214 192.084116 173.389676
83 digital_ok nn Temporal Discontinuties 163.991624 53.709995 50.841731 nan nan 144.674686 109.036247 163.991624 95.086818
82 RF_maintenance ee Temporal Discontinuties 145.773699 9.141039 35.680475 nan nan 13.793234 124.789820 4.114789 145.773699
86 RF_maintenance ee Temporal Variability 103.185568 2.434152 90.226679 nan nan 13.946841 103.185568 7.571420 68.835637
136 RF_maintenance ee Shape 85.388435 4.691449 85.388435 nan nan 18.381580 13.685091 9.252900 12.094371
179 digital_ok ee Shape 82.482400 6.584218 82.482400 nan nan 25.070794 13.579015 6.888265 18.135198
99 digital_ok ee Shape 82.350070 1.041389 82.350070 nan nan 10.634801 9.397967 10.002780 6.627923
130 digital_ok ee Shape 76.655799 5.881718 76.655799 nan nan 10.122168 11.036662 7.099242 13.323960
145 digital_ok ee Shape 75.393168 2.949062 75.393168 nan nan 7.534698 11.598997 5.733908 14.128457
92 digital_ok ee Shape 65.113841 2.499367 65.113841 nan nan 11.436083 9.031447 6.860983 13.148895
123 digital_ok ee Shape 62.438094 5.957712 62.438094 nan nan 32.676572 10.033804 2.542690 5.006685
84 RF_maintenance ee Temporal Variability 29.157516 4.629517 9.448769 nan nan 10.251751 29.157516 0.079860 -1.039747
93 RF_maintenance nn Temporal Variability 28.831034 1.792356 2.647611 nan nan 28.831034 12.527857 5.848304 3.885632
156 digital_ok ee Temporal Variability 19.279683 2.192846 2.220287 nan nan 18.551220 19.279683 2.943022 3.736550
166 digital_ok nn Shape 17.857994 17.857994 1.112161 nan nan 11.857415 -0.356780 2.737713 4.467895
157 digital_ok nn Temporal Variability 15.540995 -0.484021 0.227523 nan nan 15.540995 10.346938 6.522429 4.396073
102 digital_ok ee Temporal Variability 14.179697 5.440993 1.754635 nan nan 5.827104 14.179697 0.461791 12.135053
165 digital_ok nn Temporal Variability 14.154429 3.461204 5.518232 nan nan 14.154429 2.036665 2.243952 0.242783
186 digital_ok ee Temporal Variability 9.976772 -0.478144 0.945011 nan nan 6.475144 9.976772 6.140700 8.982142
87 digital_ok nn Temporal Variability 8.165534 3.494297 1.319095 nan nan 8.165534 4.530005 5.251410 1.382681
164 digital_ok ee Shape 7.868631 -1.860728 7.868631 nan nan 0.981094 6.546839 -1.727187 -1.095224
81 RF_maintenance ee Shape 7.227155 -1.049368 7.227155 nan nan -2.613771 3.365582 -1.460526 -3.327572
98 digital_ok nn Shape 7.225151 7.225151 1.597622 nan nan 4.245832 1.080258 1.635417 5.006812
104 digital_ok ee Shape 5.299313 -0.658948 5.299313 nan nan -2.141230 0.252265 -2.022055 -1.421722
129 digital_ok ee Temporal Variability 4.376426 2.934405 1.667970 nan nan 1.513455 4.376426 3.238207 3.246062
45 RF_maintenance ee Temporal Discontinuties 4.189152 -0.295796 1.367578 nan nan 1.713342 -1.134732 1.973065 4.189152
46 digital_ok nn Temporal Discontinuties 2.815591 0.648699 0.101964 nan nan 1.104951 -1.328688 2.815591 1.033430
100 digital_ok nn Temporal Variability 2.680412 0.075897 1.233544 nan nan 2.680412 1.539078 -0.283874 -1.755980
176 digital_ok nn Temporal Discontinuties 2.502431 0.058874 -0.058874 nan nan -0.270442 -1.628892 2.502431 1.342068
135 digital_ok nn Temporal Variability 2.485241 -2.230828 -0.646542 nan nan 2.485241 0.037235 1.275208 0.101618
119 digital_ok nn Shape 2.426657 2.426657 1.080811 nan nan 1.451745 -0.682822 -2.356953 -3.062886
101 RF_maintenance ee Shape 2.054583 -2.173885 2.054583 nan nan -1.131265 -3.566056 -0.493666 0.803170
185 digital_ok ee Temporal Discontinuties 1.998325 -0.198175 -1.270322 nan nan 0.223282 -0.037235 1.504674 1.998325
177 digital_ok ee Temporal Variability 1.316663 -2.419143 -2.043198 nan nan -1.086410 1.316663 0.409592 -0.337927
144 digital_ok nn Shape 0.542365 0.542365 -1.895188 nan nan -0.496657 -2.668974 0.196695 -0.676840
163 digital_ok ee Temporal Discontinuties -0.101618 -2.404530 -0.674961 nan nan -2.107720 -1.033248 -0.313510 -0.101618
120 digital_ok nn Shape -0.435147 -0.435147 -3.003752 nan nan -2.719489 -2.601330 -2.253539 -1.301297
In [13]:
def Flag_Bar_Chart():
    plt.figure(figsize=(8, 4), dpi=100)

    # count 
    rationales = list(binary_flags.keys())
    flags_list = np.array([binary_flags[rationale] for rationale in rationales])
    antpol_flags = [np.sum(f) for f in flags_list]
    ant_flags = [np.sum(np.array(f)[0::2] | np.array(f)[1::2]) for f in flags_list]

    # make bar chart
    plt.bar(np.arange(len(rationales)), antpol_flags, width=.7, color='yellow', ec='k', tick_label=rationales)
    for x, (nflags, nants) in enumerate(zip(antpol_flags, ant_flags)):
        plt.text(x, nflags/2, f'{nflags} Feeds on \n{nants} Antennas\nFlagged', va='center', ha='center')

    # set labels
    plt.ylabel('Antenna-Polarizations Flagged')
    plt.xlabel('Reason for Flagging')
    plt.tight_layout()

Figure 2: Flagging Rationale Summary

This bar chart summarizes the number of antenna-polarizations that are statistical outliers in each metric (though often they overlap). Some of these issues occur on both polarizations, so there are fewer unique antennas flagged for each rationale than there are ant-pols flagged, as noted by the labels.

In [14]:
Flag_Bar_Chart()
In [15]:
def Rationale_Corr_Plot():
    plt.figure(figsize=(6,6), dpi=100)
    
    # compute correlation matrix
    rationales = list(binary_flags.keys())
    flags_list = np.array([binary_flags[rationale] for rationale in rationales])
    corrs = np.corrcoef(flags_list)
    
    # plot and label correlation matrix
    plt.imshow(corrs, cmap='viridis', interpolation='nearest', origin='upper')
    for i in range(corrs.shape[0]):
        for j in range(corrs.shape[1]):
            plt.text(i, j, np.round(corrs[i, j], 3), va='center', ha='center',
                     bbox={'facecolor': 'w', 'ec': 'w', 'alpha': .75})

    # colorbar, labels, and style
    plt.yticks(range(len(rationales)), rationales)
    plt.xticks(range(len(rationales)), rationales, rotation=-45, ha='right')
    plt.gca().xaxis.tick_top()
    plt.clim([0, 1])
    plt.colorbar(fraction=0.046, pad=0.04)
    plt.tight_layout()

Figure 3: Flagging Rationale Correlations

This plot shows the probability that if a given ant-pol is flagged for some reason, it's also flagged for another reason.

In [16]:
Rationale_Corr_Plot()
In [17]:
def plot_all_spectra(spectra, mod_zs, modz_cut, overall, freqs, reason, ex_ants=[], 
                     xlabel='Frequency (MHz)', ylabel='', yscale='linear', ylim_factor=None):
    '''Helper function for plotting all spectra and showing which ones were flagged and why.''' 
    fig, axes = plt.subplots(1,2, figsize=(14,5), dpi=100)
    pols = sorted(set([bl[2] for bl in spectra]))
    for ax, pol in zip(axes, pols):
        # sort antennas into good, bad, and bad but not for this reason
        bad_here = [bl for bl in spectra if (bl[2] == pol) and mod_zs[bl] > modz_cut]
        other_bad = [bl for bl in spectra if (bl[2] == pol) and (bl[0] in ex_ants) and (bl not in bad_here)]
        good = [bl for bl in spectra if (bl[2] == pol) and (bl[0] not in ex_ants)]
        
        # plot all spectra
        l1, l2, l3 = None, None, None
        for bl in other_bad:
            l2, = ax.plot(freqs, spectra[bl], 'darkviolet', lw=.5)
        for bl in bad_here:
            l1, = ax.plot(freqs, spectra[bl], 'r', lw=.5)
        for bl in good:
            l3, = ax.plot(freqs, spectra[bl], 'grey', alpha=.5)
        l4, = ax.plot(freqs, overall[bl[2]], 'k--')
        
        ax.set_xlabel(xlabel)
        ax.set_ylabel(ylabel)
        ax.set_yscale(yscale)
        if ylim_factor is not None:
            ax.set_ylim([np.nanmin([spectra[bl] for bl in good]) / ylim_factor, 
                         np.nanmax([spectra[bl] for bl in good]) * ylim_factor])
        ax.set_title(f'Outliers in Autocorrelation {reason}: {pol}')
        ax.legend([l1, l2, l3, l4], [f'Flagged for {reason}', 'Flagged for Another Reason', 'Unflagged', 'Average Unflagged'], loc='lower right')
    plt.tight_layout()
In [18]:
overall_shape = {pol: np.nanmean([spec for bl, spec in am['spectra']['mean_spectra_normed'].items() if (bl[2] == pol) 
                                  and (bl[0] not in ex_ants)], axis=0) for pol in autos.pols()}
overall_power = {pol: np.nanmean([spec for bl, spec in am['spectra']['mean_spectra'].items() if (bl[2] == pol) 
                                  and (bl[0] not in ex_ants)], axis=0) for pol in autos.pols()}
overall_temp_var = {pol: np.nanmean([spec for bl, spec in am['spectra']['std_spectra_normed'].items() if (bl[2] == pol) 
                                     and (bl[0] not in ex_ants)], axis=0) for pol in autos.pols()}
overall_temp_diff = {pol: np.nanmean([spec for bl, spec in am['spectra']['mean_abs_diff_spectra_normed'].items() if (bl[2] == pol) 
                                      and (bl[0] not in ex_ants)], axis=0) for pol in autos.pols()}

Figure 4: Outliers in Autocorrelation Shape

This plot summarizes the spectra computed to compare to one another to find outliers in autocorrelation shape (see above for how that was computed). The mean compared to is shown as a black dashed line. Antennas in red were flagged as outliers, antennas in gray and purple were not. However, antennas in purple were flagged for some other reason, either another metric or on the other polarization. Completely flagged channels (RFI and band edges) appear as white gaps.

In [19]:
plot_all_spectra(am['spectra']['mean_spectra_normed'], am['modzs']['r2_shape_modzs'], mean_round_modz_cut, 
                 overall_shape, freqs, 'Shape', ex_ants=ex_ants, yscale='linear', ylim_factor=1.2)

Figure 5: Outliers in Autocorrelation Power

This plot summarizes the spectra computed to compare to one another to find outliers in autocorrelation amplitude (see above for how that was computed). The mean compared to is shown as a black dashed line. Antennas in red were flagged as outliers, antennas in gray and purple were not. However, antennas in purple were flagged for some other reason, either another metric or on the other polarization. Completely flagged channels (RFI and band edges) appear as white gaps.

In [20]:
plot_all_spectra(am['spectra']['mean_spectra'], am['modzs']['r2_power_modzs'], mean_round_modz_cut, 
                 overall_power, freqs, 'Power', ex_ants=ex_ants, yscale='log')

Figure 6: Outliers in Autocorrelation Temporal Variability

This plot summarizes the spectra computed to compare to one another to find outliers in autocorrelation temporal variability (as measured by a standard deviation over time; see above for how that was computed). The mean compared to is shown as a black dashed line. Antennas in red were flagged as outliers, antennas in gray and purple were not. However, antennas in purple were flagged for some other reason, either another metric or on the other polarization. Completely flagged channels (RFI and band edges) appear as white gaps.

In [21]:
plot_all_spectra(am['spectra']['std_spectra_normed'], am['modzs']['r2_temp_var_modzs'], mean_round_modz_cut,
                 overall_temp_var, freqs, 'Temporal Variability', ex_ants=ex_ants, yscale='log')

Figure 7: Outliers in Autocorrelation Temporal Discontinuities

This plot summarizes the spectra computed to compare to one another to find outliers in autocorrelation temporal discontinuities (as measured by the average absolute integration-to-integration difference over time; see above for how that was computed). The mean compared to is shown as a black dashed line. Antennas in red were flagged as outliers, antennas in gray and purple were not. However, antennas in purple were flagged for some other reason, either another metric or on the other polarization. Completely flagged channels (RFI and band edges) appear as white gaps.

In [22]:
plot_all_spectra(am['spectra']['mean_abs_diff_spectra_normed'], am['modzs']['r2_temp_diff_modzs'], mean_round_modz_cut, 
                 overall_temp_diff, freqs, 'Temporal Discontinutities', ex_ants=ex_ants, yscale='log')
In [23]:
# compute average good autocorrelations for each polarization
avg_good_autos = {pol: np.zeros(wf_shape, dtype=float) for pol in autos.pols()}
for pol in autos.pols():
    for i in range(wf_shape[0]):
        avg_good_autos[pol][i] = np.mean([np.abs(autos[bl][i, :]) for bl in autos 
                                          if (bl[0] not in ex_ants) and (bl[2] == pol)], axis=0)
    avg_good_autos[pol][am['flags']] = np.nan
In [24]:
def Avg_Auto_Plot():
    fig, axes = plt.subplots(1, 2, figsize=(14,5), dpi=100)
    for ax, pol in zip(axes, autos.pols()):
        im = ax.imshow(avg_good_autos[pol], aspect='auto', interpolation='nearest',
                       extent=[freqs[0], freqs[-1], times[-1], times[0]])
        ax.set_yticklabels(np.around(lsts[[min(max(np.searchsorted(times, t), 0), len(times) - 1) for t in ax.get_yticks()]], 2))    
        plt.colorbar(im, ax=ax)
        ax.set_title(f'Average Good Raw {pol} Autocorrelation After Flagging')
        ax.set_ylabel('LST (hours)')
        ax.set_xlabel('Frequency (MHz)')
    plt.tight_layout()

Figure 8: Average Good Autocorrelations and Flags

Here we show the waterfalls of the array-averaged autocorrelations over the night, after removing all flagged antennas. We also show the RFI mask generated between the median and mean rounds of antenna outlier detection. This is meant to show that there is little or no RFI remaining to affect the statistics.

In [25]:
Avg_Auto_Plot()

Per-Antenna Plots

In [26]:
def plot_spectra(axes, ant, spec, modzs, modz_cut, overall, reason, yscale='linear'):
    '''Helper function for plotting both antennas of a given polarization.'''
    for pol, ax in zip(autos.pols(), axes):
        bl = (ant, ant, pol)

        # plot good antennas
        for bl2 in modzs:
            if (bl2[0] not in ex_ants) and (bl2[2] == pol):
                ax.plot(freqs, spec[bl2], 'grey', lw=.5, alpha=.5)
        ax.plot(freqs, overall[pol], 'k--')

        # plot this anetnna
        color = 'r'
        alpha = .75
        if modzs[bl] >= modz_cut:
            alpha = 1
        elif bl[0] in ex_ants:
            color = 'darkviolet'
        else:
            color = 'darkgreen'
        ax.plot(freqs, spec[bl], color, alpha=alpha, label=f'{ant} {pol} (z = {np.round(modzs[bl],1)})')

        # decorate axis
        ax.set_yscale(yscale)
        ax.legend(loc=4)
        ax.set_title(f'{pol} {reason}')  
In [27]:
len(ants)
Out[27]:
65
In [28]:
def Plot_All_Auto_Metrics():
    for row, ant in enumerate(df['Ant']): 
        # print title of section
        display(HTML(f'<h2>Antenna {ant}:</h2>'))

        # print metrics
        df_row = df.loc[df['Ant'] == ant]
        df_row = df_row.style.hide_index()\
                       .set_table_styles([dict(selector="th",props=[('max-width', '70pt')])])\
                       .applymap(lambda val: f'background-color: {status_colors[val]}' if val in status_colors else '', subset=['A Priori Status']) \
                       .applymap(lambda val: 'font-weight: bold' if val in ex_ants else '', subset=['Ant']) \
                       .applymap(lambda val: 'color: red' if val in ex_ants else '', subset=['Ant']) \
                       .applymap(lambda val: 'font-weight: bold' if val > am['parameters']['mean_round_modz_cut'] else '', subset=z_score_cols) \
                       .applymap(lambda val: 'color: red' if val > am['parameters']['mean_round_modz_cut'] else '', subset=z_score_cols)
        display(HTML(df_row.render()))

        # plot spectra and waterfalls
        fig, axes = plt.subplots(2, 6, figsize=(18, 5.5), dpi=(50 * (104 / len(ants))**.5)) # this should help manage filesize

        # plot individual spectra compared to all good antennas
        plot_spectra(axes[:, 0], ant, am['spectra']['mean_spectra_normed'], am['modzs']['r2_shape_modzs'], 
                     mean_round_modz_cut, overall_shape, 'Shape', yscale='linear')
        plot_spectra(axes[:, 1], ant, am['spectra']['mean_spectra'], am['modzs']['r2_power_modzs'], 
                     mean_round_modz_cut, overall_power, 'Power', yscale='log')
        plot_spectra(axes[:, 2], ant, am['spectra']['std_spectra_normed'], am['modzs']['r2_temp_var_modzs'],
                     mean_round_modz_cut, overall_temp_var, 'Temporal Variability', yscale='linear')
        plot_spectra(axes[:, 3], ant, am['spectra']['mean_abs_diff_spectra_normed'], am['modzs']['r2_temp_diff_modzs'],
                     mean_round_modz_cut, overall_temp_diff, 'Temporal Discontinutities', yscale='linear')
        plt.tight_layout()

        # plot linear-scale waterfalls 
        for pol, ax in zip(autos.pols(), axes[:, 4]):
            bl = (ant, ant, pol)        
            im = ax.imshow(np.where(am['flags'], np.nan, autos[bl].real), 
                           aspect='auto', interpolation='nearest',
                           extent=[freqs[0], freqs[-1], times[-1], times[0]])
            ax.set_yticklabels(np.around(lsts[[min(max(np.searchsorted(times, t), 0), len(times) - 1) for t in ax.get_yticks()]], 2))
            ax.set_title(f'{pol} Waterfall (Linear Scale)')

        # plot log-scale mean-divided waterfalls
        for pol, ax in zip(autos.pols(), axes[:, 5]):
            bl = (ant, ant, pol)
            to_plot = autos[bl].real / avg_good_autos[pol]
            to_plot[am['flags']] = np.nan
            to_plot /= np.nanmean(to_plot)
            im = ax.imshow(np.log10(to_plot), aspect='auto', interpolation='nearest', vmin=-.07, vmax=.07,
                           extent=[freqs[0], freqs[-1], times[-1], times[0]])
            ax.set_yticklabels(np.around(lsts[[min(max(np.searchsorted(times, t), 0), len(times) - 1) for t in ax.get_yticks()]], 2))
            ax.set_title(f'{pol} Log(Normalized Waterfall)')

        plt.show()

        # print some whitespace
        display(HTML('<hr style="height:3px">'))

Figure 9: Per-Antenna Statistics, Spectra, and Waterfalls

Here we show the metrics for each antenna and the spectra/waterfalls that hopefully explain what led to them. The table reproduces the information from Table 1 above. The first four panels in each row clearly highlight the antenna's spectrum as it compares to the mean good antenna (black) and the distribution of good antennas (gray). Spectra in red were flagged as outliers. Spectra in purple were flagged for some other reason, either another metric or on the other polarization. Good antennas are shown in green. Completely flagged channels (RFI and band edges) appear as white gaps. In the fifth column, the waterfall of that autocorrelation is shown on a linear scale after RFI/band edge flags (white). In the sixth column, we show the log (base 10) of the same waterfall, divided by the average good antennas' waterfall of that polarization and then normalized to an average of 1.

In [29]:
Plot_All_Auto_Metrics()

Antenna 2:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
2 RF_maintenance nn Temporal Discontinuties 2114.273409 434.747897 474.186072 nan nan 784.128533 749.796613 2114.273409 1972.402185

Antenna 1:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
1 digital_maintenance nn Temporal Discontinuties 1311.205620 529.299680 533.324560 nan nan 462.830001 433.445162 1311.205620 1011.981816

Antenna 24:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
24 RF_maintenance ee Temporal Discontinuties 1262.808006 531.062547 533.230048 nan nan 257.255477 655.066687 421.164272 1262.808006

Antenna 0:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
0 digital_maintenance nn Temporal Discontinuties 1239.341529 386.076769 331.281266 nan nan 556.289011 510.911822 1239.341529 1150.166483

Antenna 12:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
12 RF_maintenance ee Temporal Discontinuties 1227.818790 458.936889 427.402072 nan nan 399.407473 474.281693 915.141199 1227.818790

Antenna 23:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
23 RF_maintenance nn Temporal Discontinuties 1012.105831 281.294592 371.828143 nan nan 402.627523 421.157687 1012.105831 979.650203

Antenna 14:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
14 digital_maintenance ee Temporal Discontinuties 1005.918902 476.830517 314.974753 nan nan 297.174015 356.167247 621.649728 1005.918902

Antenna 13:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
13 digital_maintenance ee Temporal Discontinuties 987.620856 529.987959 533.390714 nan nan 366.530437 444.024939 817.126921 987.620856

Antenna 11:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
11 digital_maintenance nn Temporal Discontinuties 882.986741 498.613135 490.080818 nan nan 346.541226 250.515666 882.986741 531.133810

Antenna 39:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
39 digital_maintenance ee Temporal Discontinuties 881.276120 531.337695 533.857976 nan nan 134.277709 482.114307 280.391022 881.276120

Antenna 127:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
127 digital_ok nn Temporal Discontinuties 600.595454 39.331387 0.034355 nan nan 508.242101 94.401801 600.595454 119.339639

Antenna 25:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
25 digital_maintenance ee Temporal Discontinuties 556.153810 517.219840 520.209054 nan nan 283.001030 268.928329 534.881701 556.153810

Antenna 121:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
121 RF_maintenance ee Shape 531.953438 122.794166 531.953438 nan nan 21.631198 27.253250 22.589585 27.197906

Antenna 116:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
116 RF_maintenance ee Shape 531.719250 111.547255 531.719250 nan nan 21.310906 30.316771 29.919520 32.308088

Antenna 73:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
73 digital_ok ee Shape 529.755837 4.499949 529.755837 nan nan 7.526088 18.811282 7.242069 25.347927

Antenna 143:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
143 digital_ok ee Shape 528.753541 72.182798 528.753541 nan nan 31.476749 24.021851 5.526966 19.250580

Antenna 26:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
26 digital_maintenance ee Shape 528.537610 523.880316 528.537610 nan nan 273.689379 144.747555 488.313535 210.718513

Antenna 187:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
187 digital_ok ee Shape 528.067346 6.997110 528.067346 nan nan 14.242670 23.986164 15.944763 16.274843

Antenna 103:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
103 digital_ok ee Shape 527.865890 1.862067 527.865890 nan nan 9.076598 12.845035 5.201319 5.543980

Antenna 184:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
184 RF_maintenance ee Shape 525.791105 3.235650 525.791105 nan nan 13.479134 16.120330 8.819277 13.891883

Antenna 112:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
112 RF_maintenance ee Temporal Discontinuties 519.116906 19.491544 39.130278 nan nan 209.422162 462.214718 126.033189 519.116906

Antenna 158:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
158 digital_ok ee Shape 519.009014 9.094384 519.009014 nan nan 20.326965 25.077673 12.460530 16.395519

Antenna 94:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
94 digital_ok ee Shape 515.134757 6.994364 515.134757 nan nan 14.481489 31.901461 14.049587 41.844616

Antenna 122:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
122 digital_maintenance ee Shape 513.241409 7.382939 513.241409 nan nan 10.746671 12.911785 7.195036 15.237563

Antenna 128:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
128 digital_ok ee Shape 508.262728 3.854949 508.262728 nan nan 185.177185 37.270363 129.634712 39.339875

Antenna 155:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
155 RF_maintenance ee Shape 504.018185 122.131750 504.018185 nan nan 30.022377 68.463956 44.094161 65.059389

Antenna 138:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
138 RF_maintenance ee Shape 458.555166 1.779760 458.555166 nan nan 12.251629 11.871141 3.084650 11.362052

Antenna 178:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
178 digital_ok ee Shape 457.487924 3.729029 457.487924 nan nan 12.722815 31.536858 7.177451 16.364962

Antenna 85:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
85 digital_ok nn Temporal Discontinuties 192.084116 11.156367 6.768437 nan nan 101.017974 85.549214 192.084116 173.389676

Antenna 83:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
83 digital_ok nn Temporal Discontinuties 163.991624 53.709995 50.841731 nan nan 144.674686 109.036247 163.991624 95.086818

Antenna 82:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
82 RF_maintenance ee Temporal Discontinuties 145.773699 9.141039 35.680475 nan nan 13.793234 124.789820 4.114789 145.773699

Antenna 86:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
86 RF_maintenance ee Temporal Variability 103.185568 2.434152 90.226679 nan nan 13.946841 103.185568 7.571420 68.835637

Antenna 136:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
136 RF_maintenance ee Shape 85.388435 4.691449 85.388435 nan nan 18.381580 13.685091 9.252900 12.094371

Antenna 179:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
179 digital_ok ee Shape 82.482400 6.584218 82.482400 nan nan 25.070794 13.579015 6.888265 18.135198

Antenna 99:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
99 digital_ok ee Shape 82.350070 1.041389 82.350070 nan nan 10.634801 9.397967 10.002780 6.627923

Antenna 130:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
130 digital_ok ee Shape 76.655799 5.881718 76.655799 nan nan 10.122168 11.036662 7.099242 13.323960

Antenna 145:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
145 digital_ok ee Shape 75.393168 2.949062 75.393168 nan nan 7.534698 11.598997 5.733908 14.128457

Antenna 92:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
92 digital_ok ee Shape 65.113841 2.499367 65.113841 nan nan 11.436083 9.031447 6.860983 13.148895

Antenna 123:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
123 digital_ok ee Shape 62.438094 5.957712 62.438094 nan nan 32.676572 10.033804 2.542690 5.006685

Antenna 84:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
84 RF_maintenance ee Temporal Variability 29.157516 4.629517 9.448769 nan nan 10.251751 29.157516 0.079860 -1.039747

Antenna 93:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
93 RF_maintenance nn Temporal Variability 28.831034 1.792356 2.647611 nan nan 28.831034 12.527857 5.848304 3.885632

Antenna 156:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
156 digital_ok ee Temporal Variability 19.279683 2.192846 2.220287 nan nan 18.551220 19.279683 2.943022 3.736550

Antenna 166:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
166 digital_ok nn Shape 17.857994 17.857994 1.112161 nan nan 11.857415 -0.356780 2.737713 4.467895

Antenna 157:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
157 digital_ok nn Temporal Variability 15.540995 -0.484021 0.227523 nan nan 15.540995 10.346938 6.522429 4.396073

Antenna 102:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
102 digital_ok ee Temporal Variability 14.179697 5.440993 1.754635 nan nan 5.827104 14.179697 0.461791 12.135053

Antenna 165:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
165 digital_ok nn Temporal Variability 14.154429 3.461204 5.518232 nan nan 14.154429 2.036665 2.243952 0.242783

Antenna 186:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
186 digital_ok ee Temporal Variability 9.976772 -0.478144 0.945011 nan nan 6.475144 9.976772 6.140700 8.982142

Antenna 87:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
87 digital_ok nn Temporal Variability 8.165534 3.494297 1.319095 nan nan 8.165534 4.530005 5.251410 1.382681

Antenna 164:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
164 digital_ok ee Shape 7.868631 -1.860728 7.868631 nan nan 0.981094 6.546839 -1.727187 -1.095224

Antenna 81:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
81 RF_maintenance ee Shape 7.227155 -1.049368 7.227155 nan nan -2.613771 3.365582 -1.460526 -3.327572

Antenna 98:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
98 digital_ok nn Shape 7.225151 7.225151 1.597622 nan nan 4.245832 1.080258 1.635417 5.006812

Antenna 104:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
104 digital_ok ee Shape 5.299313 -0.658948 5.299313 nan nan -2.141230 0.252265 -2.022055 -1.421722

Antenna 129:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
129 digital_ok ee Temporal Variability 4.376426 2.934405 1.667970 nan nan 1.513455 4.376426 3.238207 3.246062

Antenna 45:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
45 RF_maintenance ee Temporal Discontinuties 4.189152 -0.295796 1.367578 nan nan 1.713342 -1.134732 1.973065 4.189152

Antenna 46:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
46 digital_ok nn Temporal Discontinuties 2.815591 0.648699 0.101964 nan nan 1.104951 -1.328688 2.815591 1.033430

Antenna 100:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
100 digital_ok nn Temporal Variability 2.680412 0.075897 1.233544 nan nan 2.680412 1.539078 -0.283874 -1.755980

Antenna 176:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
176 digital_ok nn Temporal Discontinuties 2.502431 0.058874 -0.058874 nan nan -0.270442 -1.628892 2.502431 1.342068

Antenna 135:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
135 digital_ok nn Temporal Variability 2.485241 -2.230828 -0.646542 nan nan 2.485241 0.037235 1.275208 0.101618

Antenna 119:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
119 digital_ok nn Shape 2.426657 2.426657 1.080811 nan nan 1.451745 -0.682822 -2.356953 -3.062886

Antenna 101:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
101 RF_maintenance ee Shape 2.054583 -2.173885 2.054583 nan nan -1.131265 -3.566056 -0.493666 0.803170

Antenna 185:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
185 digital_ok ee Temporal Discontinuties 1.998325 -0.198175 -1.270322 nan nan 0.223282 -0.037235 1.504674 1.998325

Antenna 177:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
177 digital_ok ee Temporal Variability 1.316663 -2.419143 -2.043198 nan nan -1.086410 1.316663 0.409592 -0.337927

Antenna 144:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
144 digital_ok nn Shape 0.542365 0.542365 -1.895188 nan nan -0.496657 -2.668974 0.196695 -0.676840

Antenna 163:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
163 digital_ok ee Temporal Discontinuties -0.101618 -2.404530 -0.674961 nan nan -2.107720 -1.033248 -0.313510 -0.101618

Antenna 120:

Ant A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
120 digital_ok nn Shape -0.435147 -0.435147 -3.003752 nan nan -2.719489 -2.601330 -2.253539 -1.301297

In [30]:
from hera_qm import version
print(version.construct_version_info())
{'version': '1.0', 'git_origin': 'https://github.com/HERA-Team/hera_qm.git', 'git_hash': 'a5eb3529b7c1419f4c9d14038de2994389b36826', 'git_description': 'v1.1-369-ga5eb352', 'git_branch': 'master'}
In [ ]: